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GAO 

^^^^^^^^^Accountability  *  Integrity  *  Reliability _ 

United  States  General  Accounting  Office 
Washington,  DC  20548 


January  24,  2002 

The  Honorable  John  Boehner 

Chairman,  Committee  on  Education  and  the  Workforce 
House  of  Representatives 

The  Honorable  Mike  Castle 
Chairman,  Subcommittee  on  Education  Reform 
Committee  on  Education  and  the  Workforce 
House  of  Representatives 

The  Honorable  Pete  Hoekstra 
Chairman,  Subcommittee  on  Select  Education 
Committee  on  Education  and  the  Workforce 
House  of  Representatives 

The  Congress  and  the  administration  have  emphasized  the  importance  of  a 
scientifically  sound  research  base  to  improve  teaching  and  learning  in  the 
nation’s  schools.  To  develop  this  research  base  and  promote  the  use  of 
research-based  practices,  the  Department  of  Education  (Education)  is 
charged  with  sponsoring  research  and  developing  and  disseminating 
research-based  activities,  such  as  technical  assistance.  Education 
administers  over  a  thousand  grants,  cooperative  agreements,^  and 
contracts  that  fund  educational  research,  development  of  materials,  new 
methods  of  instruction  and  practices  in  teaching,  dissemination  of 
research  results,  and  technical  assistance  activities.  Some  of  these  grants 
and  contracts  are  administered  through  the  Office  of  Educational 
Research  and  Improvement  (OERI),  but  others  are  administered 
elsewhere  in  Education. 

The  number  of  programs  these  grants,  cooperative  agreements,  and 
contracts  support,  as  well  as  changes  in  the  laws  authorizing  them,  have 
raised  many  questions  about  the  activities  the  programs  perform,  the 
extent  to  which  the  programs  collaborate  and  coordinate  with  each  other 
and  Education,  and  the  usefulness  of  the  evaluations  that  have  assessed 
their  performance. 


cooperative  agreement  is  a  type  of  grant  used  by  agencies  when  substantial  involvement 
and  interaction  is  expected  of  both  the  agency  and  the  recipient  in  carrying  out  the 
activities  in  the  agreement. 
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Consequently,  you  asked  us  to  conduct  a  series  of  studies  on 
Education-funded  research,  including  this  examination  of  three  programs: 
Research  and  Development  Centers  (R&D  Centers),  Regional  Educational 
Laboratories  (Regional  Labs),  and  Regional  Comprehensive  Assistance 
Centers  (Comprehensive  Centers).  Specifically,  we  agreed  to  answer: 

1.  To  what  extent  are  the  activities  of  the  R&D  Centers,  Regional  Labs, 
and  Comprehensive  Centers  consistent  with  their  legislative  mandates 
and  to  what  extent  can  Education  shape  and  control  these  activities  to 
support  its  research  agenda? 

2.  To  what  extent  do  the  R&D  Centers,  Regional  Labs,  and 
Comprehensive  Centers  collaborate  and  coordinate  with  each  other 
and  with  Education  and  what  factors  foster  or  hinder  collaboration 
and  coordination? 

3.  To  what  extent  are  Education’s  practices  for  assessing  the  R&D 
Centers,  Regional  Labs,  and  Comprehensive  Centers  useful  for 
evaluating  their  performance? 

To  answer  these  questions,  we  sent  questionnaires  to  the  12  R&D  Centers, 
10  Regional  Labs,  and  15  Comprehensive  Centers;  reviewed  documents 
from  the  R&D  Centers,  Regional  Labs,  Comprehensive  Centers,  and 
Education;  examined  relevant  statutes;  and  interviewed  Education 
officials.  We  also  conducted  14  site  visits  to  R&D  Centers,  Regional  Labs, 
and  Comprehensive  Centers  to  gather  illustrative  examples  and  to  more 
fully  understand  the  processes  involved  in  their  funding  and  operations.  In 
addition,  we  examined  the  standards  Education  uses  to  assess  R&D 
Centers  and  Regional  Labs  and  the  most  recent  Education-funded 
assessments  of  the  R&D  Centers,  Regional  Labs,  and  Comprehensive 
Centers.  We  performed  our  work  between  January  and  December  2001  in 
accordance  with  generally  accepted  government  auditing  standards.  We 
did  not  independently  verify  survey  information  reported  by  the  R&D 
Centers,  Regional  Labs,  and  Comprehensive  Centers. 


Results  in  Brief 


Activities  conducted  by  the  R&D  Centers,  Regional  Labs,  and 
Comprehensive  Centers  reflect  their  legislative  mandates  and  support 
Education’s  research  agenda  to  various  degrees.  Because  statutes  define 
different  missions  and  activities  for  these  programs,  the  amount  and  focus 
of  the  research  and  other  research-based  activities  they  support  varies. 

For  example,  law  mandates  that  R&D  Centers  engage  in  national  research, 
development,  and  dissemination  activities.  In  a  manner  consistent  with 
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this  mandate,  in  fiscal  year  2000  they  reported  spending  87  percent  of  the 
$35.5  million  in  core  funding  they  received  from  OERl  on  these  activities. 
(See  fig.  1.) 


Figure  1 :  R&D  Centers  Reported  Spending  Most  Core  Funds  from  OERi  on 
Research 
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In  contrast,  legislation  authorizes  the  Comprehensive  Centers  to  provide 
technical  assistance  and  training  to  help  state  and  local  educational 
agencies  implement  federal  programs  established  by  the  Elementary  and 
Secondary  Education  Act  (ESEA).  Accordingly,  in  fiscal  year  2000, 
Comprehensive  Centers  reported  spending  83  percent  of  the  $28.6  million 
of  their  core  funds  from  Office  of  Elementary  and  Secondary  Education 
(OESE)  on  technical  assistance.  (See  fig.  2.) 
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Figure  2:  Comprehensive  Centers  Reported  Spending  Most  Core  Funds  from  OESE 
on  Technicai  Assistance 
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The  law  requires  Regional  Labs  to  conduct  a  wide  range  of  activities  that 
reflects  primarily  the  needs  of  the  regions  they  serve.  In  line  with  this 
requirement,  in  fiscal  year  2000  the  Regional  Labs  reported  spending  fairly 
equal  amounts  as  compared  to  the  R&D  Centers  and  Comprehensive 
Centers  of  their  $65.2  milhon  in  core  funds  from  OERI  on  research, 
development,  dissemination  and  technical  assistance  activities.  (See  fig. 

3.) 
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Figure  3:  Regional  Labs  Reported  Spending  Core  Funds  from  OERI  on  a  Variety  of 
Activities 
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To  a  large  extent,  Education  shapes  the  priorities  that  guide  the  research 
conducted  by  the  R&D  Centers  and  targets  the  technical  assistance 
provided  by  the  Comprehensive  Centers  through  requirements  in 
agreements  with  these  entities.  However,  Education  has  limited  control 
over  the  activities  of  the  Regional  Labs  because  regional  governing  boards, 
mandated  by  the  legislation  authorizing  Regional  Labs,  establish  regional 
priorities  and  are  responsible  for  directing  the  Regional  Labs  in  fulfilling 
the  terms  of  their  contracts  with  Education.  Therefore,  the  Regional  Labs 
are  unlike  most  federal  education  programs  because  neither  the  federal 
government  nor  state  governments  have  oversight  responsibihty  for  their 
programs.  As  the  Congress  reauthorizes  the  Regional  Labs,  it  may  wish  to 
consider  giving  Education  responsibihty  for  the  agenda  of  the  Regional 
Labs  and  the  quality  of  the  products  and  services  they  produce  or  giving 
states  the  responsibility  by  providing  these  funds  to  each  state  for 
subsequent  distribution. 

The  R&D  Centers,  Regional  Labs,  and  Comprehensive  Centers  reported 
collaborating  and  coordinating  with  each  other  and  Education  and  cited  a 
variety  of  factors  that  facilitated  and  hindered  such  activities.  These 
programs  provided  a  variety  of  examples  of  collaboration  and 
coordination.  They  said  that  they  were  most  likely  to  engage  in  these 
activities  when  they  shared  a  common  interest  in  a  specific  student 
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population,  such  as  English  language  learners,  or  in  a  specific  topic,  such 
as  assessment.  Relationships  between  the  R&D  Centers,  Regional  Labs, 
and  Comprehensive  Centers  facilitated  collaboration  and  coordination. 
Moreover,  Education  played  a  proactive  role  in  encouraging  such  activities 
by  requiring  joint  activities  such  as  conferences  as  part  of  their  funding 
agreements,  and  by  identifying  areas  where  collaboration  would  be 
beneficial. 

Current  evaluation  practices  for  assessing  the  R&D  Centers,  Regional 
Labs,  and  Comprehensive  Centers  have  provided  only  limited  information 
about  the  performance  of  these  organizations  and  have  not  been  useful  for 
making  future  funding  decisions.  The  law  requires  that  Education  use  peer 
review  a  process  that  relies  on  knowledgeable  individuals  to  make 
independent  assessments  of  research’s  technical  and  scientific  merit  to 
evaluate  OERl  activities,  including  those  undertaken  by  the  R&D  Centers 
and  Regional  Labs.  Although  peer  review  is  well  accepted  and  widely  used 
throughout  the  government  to  assess  the  merit  of  research  proposals  and 
the  scientific  soundness  of  research,  it  does  not  directly  assess  research 
usefulness,  outcomes,  or  effects.  Moreover,  Education’s  procedures  for 
peer  reviews  had  a  potential  for  bias  and  were  cumbersome,  which  limited 
the  usefulness  of  their  findings.  Finally,  peer  review  is  of  limited  value  for 
Regional  Labs  because  most  of  their  activities  do  not  involve  research. 
With  regard  to  the  Comprehensive  Centers,  Education’s  2000  evaluation  of 
the  Comprehensive  Centers  provided  useful  information  about  the 
Comprehensive  Centers  as  a  network  but  limited  information  on  the 
performance  of  individual  centers.  Thus,  the  information  from  the 
evaluation  could  not  be  used  to  inform  decisions  pertaining  to  funding  of 
individual  Comprehensive  Centers  or  to  improve  ongoing  practices. 
Because  of  the  shortcomings  of  the  evaluations  of  R&D  Centers,  Regional 
Labs,  and  Comprehensive  Centers,  we  are  suggesting  that  the  Congress 
consider  directing  Education  to  use  other  accepted  evaluation  techniques 
to  assess  applied  research,  development,  dissemination,  and  technical 
assistance  activities.  Moreover,  we  are  recommending  to  Education  that  it 
revise  its  peer  review  standards  to  allow  for  division  of  labor  and  greater 
concentration  on  assessing  the  quality  of  projects,  services  and  products, 
rather  than  reviewing  procedural  materials.  We  are  also  recommending 
that  it  design  future  evaluations  of  the  Comprehensive  Centers  to  provide 
information  on  individual  centers. 


Page  6 


GAO-02-190  Education  Research 


Background 


The  R&D  Centers,  Regional  Labs,  and  Comprehensive  Centers  share 
responsibility  with  other  programs  created  by  the  Congress  for  education 
research,  research-based  activities,  and  technical  assistance.^  Many  of 
these  programs  are  located  in  six  different  offices  throughout  Education. 
(See  fig.  4.) 


^Education  funds  many  additional  research  and  support  entities,  including  Special 
Education  and  Early  Childhood  Regional  Resource  Centers,  Eisenhower  Regional  Math  and 
Science  Consortia,  and  National  Centers  for  Vocational  Education. 
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Figure  4:  Education  Offices  with  Responsibiiity  for  Education  Research,  Research-Based  Activities,  and  Technical  Assistance 
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For  example,  the  Individuals  with  Disabilities  Education  Act  established  a 
special  education  research  and  innovation  program  as  well  as  technical 
assistance  centers  to  improve  services  and  results  for  children  with 
disabilities.  The  Rehabilitation  Act  established  a  National  Institute  for 
Disability  and  Rehabilitation  Research.  The  Office  of  Special  Education 
and  Rehabilitative  Services  administers  these  programs.  The  amount  of 
funding  received  by  these  programs  in  fiscal  year  2000  ranged  from 
$70,000  for  11  American  Overseas  Research  Centers  to  $86.5  million  for 
the  National  Institute  on  Disability  and  Rehabilitation  Research. 

OERI  is  Education’s  lead  office  for  educational  research  and  development. 
Its  goals  are  to  promote  quality  and  equity  in  education  by  funding 
research;  developing  new  learning  materials,  teaching  techniques,  and 
methods  of  organizing  schools;  demonstrating  and  evaluating  promising 
educational  practices;  disseminating  research-based  information;  and 
collecting  data  related  to  schools  in  the  United  States  and  other  nations. 
Unlike  other  Education  offices,  OERFs  activities  span  all  grade  levels, 
from  preschool  through  adult  education,  and  all  major  content  areas  of 
instruction. 

The  R&D  Centers,  established  in  the  1960s  to  increase  fundamental 
knowledge  in  education,  are  administered  by  OERI.  Over  the  years, 
legislative  changes  have  repositioned  their  placement  in  Education.  Most 
recently,  the  Educational  Research,  Development,  Dissemination  and 
Improvement  Act  of  1994  reorganized  OERI,  implementing  measures  that 
changed  the  way  the  R&D  Centers  related  to  Education.  First,  the  act 
created  the  National  Educational  Research  Policy  and  Priorities  Board  to 
work  with  the  Assistant  Secretary  of  OERI  to  establish  a  long-term 
national  agenda  for  research,  development,  and  dissemination  activities. 
Unlike  previous  boards,  this  board  was  charged  with  improving  research 
priorities  and  developing  standards  for  evaluating  OERI  research, 
including  that  done  by  the  R&D  Centers.  Second,  the  act  established  five 
national  research  institutes^  within  OERI,  each  with  its  own  research 
focus,  and  placed  the  R&D  Centers,  as  well  as  field-initiated  studies  and 
other  research-related  programs,  under  the  appropriate  institute.  The 
Congress  is  expected  to  start  the  reauthorization  of  OERI  in  2002;  the  last 


^The  5  OERI  institutes  are  relatively  small.  Each  employed  between  8  to  14  professional 
and  support  staff.  In  contrast,  the  National  Institute  of  Mental  Health,  one  of  the 
Department  of  Health  and  Human  Services  19  institutes,  employed  over  400  staff  at  its 
headquarters  office. 
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reauthorization  was  in  1994  and  was  intended  to  be  on  a  5-year  cycle. 
Figure  5  shows  the  locations  of  the  R&D  Centers. 


Figure  5:  Locations  of  the  R&D  Centers 
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Legislation  has  changed  the  focus  of  the  Regional  Labs  over  time. 
Originally  established  to  resemble  the  labs  funded  by  the  Atomic  Energy 
Commission,  such  as  the  Los  Alamos  National  Laboratory,  the  Congress 
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created  the  Regional  Labs  in  1965  to  conduct  long-term  activities  to 
address  national  educational  problems.  However,  funding  to  support  these 
activities  was  never  made  available,  limiting  the  scope  of  the  Regional 
Labs  to  smaller-scale  projects.  Further,  in  the  late  1970s,  the  Congress’s 
negative  reactions  to  federally  supported  curriculum  projects  prompted 
Education  and  the  Regional  Labs  to  discontinue  all  large-scale  nationally 
oriented  curriculum  projects.  As  a  result.  Regional  Labs  developed  an 
increasingly  regional  agenda.  In  1994,  the  Congress  gave  the  governing 
board  of  each  Regional  Lab  sole  responsibility  for  determining  if  the 
Regional  Lab  fulfilled  the  terms  of  its  contract  with  Education  and  its 
regional  agenda.  The  Congress  mandated  that  each  governing  board 
reflects  a  balanced  representation  of  states  in  the  region,  as  well  as 
interests  and  concerns  of  regional  constituencies,  including  teachers  and 
education  researchers.  Figure  6  shows  the  states  included  in  the  regions  of 
the  10  Regional  Labs. 
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Figure  6:  Regions  of  the  Regionai  Labs 
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■PREL  includes:  American  Samoa,  Commonwealth  of  the  Northern  Mariana  Islands,  Federated 
States  of  Micronesia  (Chuuk,  Kosrae,  Pohnpei,  and  Yap),  Guam,  Hawaii,  Republic  of  Marshall 
Islands,  and  the  Republic  of  Palau. 
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Created  in  1994,  the  Comprehensive  Centers  were  established  more 
recently  than  the  R&D  Centers  and  the  Regional  Labs.  The  Improving 
America’s  Schools  Act  of  1994  consolidated  the  functions  of  48  categorical 
technical  assistance  centers'*  that  supported  programs  authorized  under 
the  ESEA,  including  Title  1,  Migrant  Education,  and  Indian  Education,  into 
15  Comprehensive  Centers.  The  Congress  created  the  Comprehensive 
Centers  to  support  comprehensive,  cross-program  assistance  as 
envisioned  in  the  law,  and  placed  them  under  the  Office  of  Elementary  and 
Secondary  Education  (OESE)  and  the  Office  of  Bilingual  Education  and 
Minority  Languages  Affairs  (OBEMLA).  Figure  7  show  the  states  included 
in  the  regions  served  by  the  Comprehensive  Centers. 


'’Categorical  programs  typically  permit  funds  to  be  used  only  for  specific,  narrowly  defined 
purposes  and  populations,  such  as  migrant,  Indian,  or  economically  disadvantaged 
children. 
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Figure  7:  Regions  of  the  Comprehensive  Centers 


■Region  XV  includes:  American  Samoa,  Commonwealth  of  the  Northern  Mariana  Islands,  Federated 
States  of  Micronesia  (Chuuk,  Kosrae,  Pohnpei,  and  Yap),  Guam,  Hawaii,  Republic  of  Marshall 
Islands,  and  the  Republic  of  Palau. 
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The  amount  of  resources  allocated  to  the  R&D  Centers,  Regional  Labs,  and 
Comprehensive  Centers  differs,  but  the  overall  investment  in  these 
programs  is  modest.  For  example,  to  operate  in  fiscal  year  2000,  each  R&D 
Center  received  from  $1.5  million  to  $6.6  million,  each  Regional  Lab 
received  from  $3.8  million  to  $8.6  million,  while  each  Comprehensive 
Center  received  from  $0.9  million  to  $2.7  million.  The  core  budget  for  all 
these  programs  totaled  about  $130  million.  Table  1  shows  the  key  features 
of  the  R&D  Centers,  Regional  Labs,  and  Comprehensive  Centers. 


Table  1 :  Key  Features  of  the  R&D  Centers,  Regional  Labs,  and  Comprehensive  Centers 

Program 

Responsible 

Education 

office 

FY  2000 
core 
funding’ 

FY  2000  average  and  range 
of  funding  per  unit  to 
support  core  program 

Allowable 

award 

recipients 

R&D  Centers 

Office  of  Educational 
Research  and  Improvement 

$35.5  million*’ 

Average:  $3  million 

Range:  $1.5  million  to  $6.6 
million 

University  partnerships 
consisting  of  universities  and 
other  not-for-profit 
organizations 

Regional 

Labs 

Office  of  Educational 
Research  and  Improvement 

$65.2  million*’ 

Average:  $6.5  million 

Range:  $3.8  million  to  $8.6 
million 

Not-for-profit  organizations 
and  universities 

Comprehensive 

Centers 

Office  of  Elementary  and 
Secondary  Education  and 
Office  of  Bilingual  Education 
and  Minority  Languages 
Affairs 

$28.6  million 

Average:  $1 .9  million 

Range:  $0.9  million  to  $2.7 
million 

Public  or  private  not-for-profit 
organizations,  universities, 
and  consortia  of  these 
institutions 

"The  core  funding  figures  do  not  include  supplemental  funds  that  R&D  Centers,  Regional  Labs  and 
Comprehensive  Centers  may  have  received  during  fiscal  year  2000  from  Education  or  other  agencies 
for  special  activities. 

Vhese  figures  are  based  on  amounts  reported  to  us  by  R&D  Center,  Regional  Lab,  and 
Comprehensive  Center  officials.  Education,  hov\/ever,  provided  somewhat  different  figures,  reporting 
$34.9  million  in  core  funding  for  the  R&D  Centers  and  $65  million  for  the  Regional  Labs. 


Because  of  the  importance  of  education  research,  research-based 
activities  and  technical  assistance  in  improving  schools,  many  studies  have 
focused  on  the  R&D  Centers,  Regional  Labs,  and  Comprehensive  Centers. 
Education  has  funded  various  assessments  of  the  R&D  Centers  and  the 
Regional  Labs  and  has  recently  conducted  an  evaluation  of  the 
Comprehensive  Centers.  In  addition,  the  R&D  Centers  and  the  Regional 
Labs  have  been  studied  and  discussed  by  numerous  independent 
organizations,  including  the  National  Academy  of  Sciences,  the  Brookings 
Institution,  the  RAND  Corporation,  and  the  National  Educational  Research 
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Policy  and  Priorities  Board. Several  studies,  including  the  1992  National 
Academy  of  Sciences  report  and  a  2000  Brookings  Institute  report,  have 
concluded  that  the  funds  available  to  OERl  to  support  research  have  been, 
and  continue  to  be,  insufficient  to  support  long-term,  large-scale  efforts. 


Activities  of  the  Three 
Programs  Reflect 
Legislative  Mandates 
and  Support 
Education’s  Agenda, 
but  Education  Has 
Limited  Control  over 
Regional  Labs 


Laws  define  different  missions  and  roles  for  the  R&D  Centers,  Regional 
Labs,  and  Comprehensive  Centers,  and  these  differences  are  reflected  in 
how  these  programs  spend  their  money.  R&D  Centers  focus  on  national 
research  priorities,  such  as  student  assessment.  Although  both  Regional 
Labs  and  Comprehensive  Centers  have  a  regional  orientation.  Regional 
Labs  focus  on  meeting  the  needs  of  the  regions.  Comprehensive  Centers 
focus  on  assisting  customers  in  their  regions  implement  federal  education 
agendas,  such  as  helping  to  close  the  achievement  gaps  for  federally 
targeted  groups  like  disadvantaged  students.  Education  uses  funding 
documents  and  program  oversight  to  direct  and  prioritize  the  activities  of 
the  R&D  Centers,  Regional  Labs,  and  Comprehensive  Centers  and  shape 
the  agendas  of  the  R&D  Centers  and  Comprehensive  Centers.  Unlike  R&D 
Centers  and  Comprehensive  Centers,  the  Regional  Labs  have  governing 
boards.  Because  these  governing  boards  determine  regional  agendas  and 
oversee  Regional  Lab  activities.  Education  has  limited  ability  to  shape  the 
agendas  of  Regional  Labs  or  ensure  accountability  for  their  products  and 
services. 


^The  National  Academy  of  Sciences,  Center  for  Education,  Committee  on  Scientific 
Principles  in  Education  Research,  Science,  Evidence,  and  Inference  in  Education:  Report 
of  a  Workshop  (Washington,  D.C.:  National  Academy  Press,  2001);  The  National  Academy 
of  Sciences,  Committee  on  Scientific  Principles  in  Education  Research,  Research  and 
Educational  Reform:  Roles  for  the  Office  of  Educational  Research  and  Improvement 
(Washington,  D.C.:  National  Academy  Press,  1992);  The  Brookings  Institution,  Can  We 
Make  Education  Policy  on  the  Basis  of  Evidence?  What  Constitutes  High  Quality 
Education  Research  and  How  Can  It  Be  Incorporated  Into  Policymaking?”  (Washington, 
D.C.:  1999);  RAND  Corporation,  Setting  Standards  for  Education  Research  (forthcoming); 
National  Educational  Research  Policy  and  Priorities  Board,  U.  S.  Department  of  Education, 
Investing  in  Learning  :  A  Policy  Statement  with  Recommendations  on  Research  in 
Education  (Washington,  D.C.:  1999)  and  A  Blueprint  for  Progress  in  American  Education 
(Washington,  D.C.:  2000). 
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Laws  Mandate  Missions 
and  Activities  of  the  R&D 
Centers,  Regional  Labs, 
and  Comprehensive 
Centers 


The  Congress  created  a  separate  primary  focus  for  the  R&D  Centers, 
Regional  Labs,  and  Comprehensive  Centers  and  gave  them  the 
responsibility  of  performing  specific  activities.  Because  of  differences  in 
their  mandates,  the  R&D  Centers,  Regional  Labs,  and  Comprehensive 
Centers  have  different  roles  in  supporting  Education’s  research  agenda. 
The  Educational  Research,  Development,  Dissemination,  and 
Improvement  Act  of  1994  places  the  R&D  Centers  under  education 
research  institutes  in  OERl,  each  of  which  addresses  a  specific  content 
area,  and  requires  them  to  carry  out  the  purposes  for  which  the  institutes 
were  created  by  conducting  research  and  development.  In  contrast,  rather 
than  requiring  the  Regional  Labs  to  address  a  particular  content  area,  the 
act  requires  them  to  use  research-based  knowledge  to  address  the  issues 
in  the  regions  they  serve  and  assigns  them  an  expansive  array  of  activities 
to  perform.  Like  the  Regional  Labs,  the  Comprehensive  Centers  have 
extensive  mandates  that  require  the  Comprehensive  Centers  to  focus  on 
certain  customers  by  giving  priority  to  schools  with  schoolwide®  programs 
and  the  highest  number  of  children  in  poverty.  The  law,  however,  allows 
Education  to  guide  the  general  direction  of  the  mandated  activities.  Table 
2  provides  the  primary  focus  of  the  R&D  Centers,  Regional  Labs,  and 
Comprehensive  Centers  and  summarizes  their  mandated  activities. 


®Schoolwide  programs  combine  resources  from  various  Education  programs,  such  as  those 
authorized  by  the  Individuals  with  Disabilities  Education  Act  and  the  Safe  and  Drug  Free 
Schools  and  Communities  Act,  to  enhance  teaching  and  learning  for  all  students  in  a 
school. 
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Table  2:  Primary  Focus  and  Mandated  Activities  of  the  R&D  Centers,  Regional  Labs,  and  Comprehensive  Centers 


Program 

R&D  Centers 


Regional 

Labs 


Comprehensive 

Centers 


Primary  focus 

Address  nationai  research 
priorities  on  specified  content 

areas  or  student  groups _ 

Serve  needs  of  geographicai 
regions  regarding  the 
implementation  of  broad-based 
systemic  school  improvement 
strategies 


Provide  technical  assistance  and 
training  related  to  the 
administration  and  implementation 
of  programs  authorized  under 
ESEA 


Mandated  activities 

Conduct  basic  research,  applied  research,  and  dissemination,  which  may 
also  include  development 


•  Conduct  applied  research  projects 

•  Develop  and  disseminate  educational  research  products 

•  Develop  a  plan  for  identifying  and  serving  the  needs  of  its  region 

•  Serve  the  educational  development  needs  of  the  region 

•  Facilitate  communication  between  education  experts,  school  officials, 
teachers,  parents,  and  librarians 

•  Provide  support,  training,  and  technical  assistance 

•  Collaborate  and  coordinate  with  other  technical  assistance  providers 

•  Bring  teams  of  experts  together  to  develop  and  implement  school 
improvement  plans 

•  Collaborate  with  the  OERI  institutes 

•  Consult  with  state  educational  agencies  and  libraries 

•  Develop  strategies  to  use  schools  as  components  in  reforming  education 
and  reviving  rural  communities 

•  Report  and  disseminate  information  on  overcoming  obstacles  faced  by 
rural  schools 

•  Identify  successful  practices  that  have  been  developed  by  the  Regional 

Labs  or  other  educational  entities  in  the  region _ 

•  Provide  support,  training,  and  assistance  to  state  and  local  educational 
agencies,  tribal  divisions  of  education,  and  other  recipients  of  Education 
funds 

•  Improve  the  quality  of  instruction,  curricula,  assessments,  and  other 
aspects  of  school  reform 

•  Implement  effective  schoolwide  programs 

•  Meet  the  needs  of  children  served  by  programs  funded  by  Education 

•  Implement  high  quality  professional  development 

•  Improve  the  quality  of  bilingual  education 

•  Create  safe  and  drug  free  environments 

•  Implement  educational  applications  of  technology 

•  Evaluate  programs 

•  Expand  the  involvement  of  parents  in  the  education  of  their  children 

•  Reform  schools,  school  systems,  and  the  governance  and  management 
of  schools 

•  Meet  the  special  needs  of  schools  and  children  in  urban  and  rural  areas 

•  Provide  technical  assistance,  and  coordinate  and  cooperate  with 
Regional  Labs,  Eisenhower  regional  consortia,  literacy  centers,  and 
other  entities  engaged  in  research,  development,  dissemination,  and 
technical  assistance 


The  spending  patterns  of  the  R&D  Centers,  Comprehensive  Centers,  and 
Regional  Labs  reflect  their  mandates  and  missions.  For  example,  the 
Congress  authorized  the  R&D  Centers  to  conduct  research  and 
development  in  order  to  increase  fundamental  knowledge  of  central  issues 
in  education.  To  support  this  mission,  they  reported  spending  73  percent 
of  the  $35.5  million  in  core  funding  they  received  from  OERI  in  fiscal  year 
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2000  on  research  and  an  additional  14  percent  on  development  and 
dissemination/  The  Congress  created  the  Comprehensive  Centers  to  help 
state  educational  agencies,  school  districts,  and  schools  within  an 
assigned  region  implement  federal  elementary  and  secondary  school 
programs  by  providing  technical  assistance  and  training.  Accordingly,  the 
Comprehensive  Centers  reported  spending  most  of  their  fiscal  year  2000 
$28.6  million  core  funds  from  OESE  83  percent  on  technical  assistance. 
Regional  Labs  were  authorized  to  conduct  a  wide  range  of  research-based 
activities,  including  applied  research,*^  development,  dissemination,  and 
technical  assistance.  Their  spending  reflected  these  purposes:  the  Regional 
Labs  reported  spending  25  percent  of  their  fiscal  year  2000  $65.2  million 
core  funds  from  OERl  on  research,  17  percent  on  development,  16  percent 
on  dissemination,  and  21  percent  on  technical  assistance. 

Most  R&D  Centers,  three  Regional  Labs,  and  two  Comprehensive  Centers 
used  funds  from  other  organizations,  including  other  federal  agencies, 
state  and  local  educational  agencies,  and  foundations,  to  expand  work 
they  were  performing  for  Education.  For  example,  one  R&D  Center,  the 
National  Center  for  Postsecondary  Improvement,  used  funding  from  the 
Pew  Charitable  Trust  to  extend  the  scope  of  a  study  examining  the  impact 
of  state  and  university  admission  standards  on  secondary  schools  and 
students.  Similarly,  10  R&D  Centers  reported  that  they  leveraged 
additional  money  from  other  federal  and  state  agencies  and  not-for-profit 
organizations  to  promote  Education’s  research  agenda.  Two  of  these  R&D 


^We  pretested  the  survey  with  officials  of  the  R&D  Centers  and  Comprehensive  Centers. 
These  officials  agreed  that  the  general  categories  of  activities — research,  dissemination, 
technical  assistance,  collaboration,  development  and  evaluation — were  sufficiently 
distinct.  However,  in  follow-up  interviews  with  respondents,  some  noted  the  difficulty  in 
separating  certain  interrelated  activities  from  one  another.  For  example,  respondents 
reported  difficulty  in  separating  technical  assistance  from  dissemination,  since  both  might 
take  place  during  the  course  of  one  activity.  Consequently,  percentages  reported  are 
estimates. 

*The  terms  “basic  research”  and  “applied  research”  are  found  in  the  authorizing  legislation 
for  the  R&D  Centers  and  “applied  research”  is  found  in  the  authorizing  legislation  for  the 
Regional  Labs,  but  the  legislation’s  definition  for  research  does  not  distinguish  between 
basic  and  applied  research.  Similarly,  the  congressionally  mandated  National  Education 
Research  Policies  and  Priorities  Board  discusses  these  terms  in  Investing  in  Learning  :  A 
Policy  Statement  with  Recommendations  on  Research  in  Education  (Washington,  D.C.: 
1999)  but  gives  no  formal  definitions.  Fiirther,  the  Board  concluded  that  Education 
conducts  no  basic  research.  Rather,  such  research  is  conducted  in  other  federal  agencies, 
most  notably  the  National  Institute  for  Child  Health  and  Human  Development,  the  Office  of 
Naval  Research,  and  the  National  Science  Foundation.  Education  concentrates  on  applied 
research  and,  according  to  the  Board,  is  contributing  some  of  the  important  applied 
research  aimed  at  comprehensive  or  standards-based  reform,  testing,  and  assessment. 
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Centers  reported  receiving  more  funding  from  these  other  sources  of 
funding  than  from  OERI.  Three  Regional  Labs  also  reported  receiving 
funds  from  non-OERl  sources,  with  these  funds  composing  between 
1  to  27  percent  of  their  budgets.  Two  Comprehensive  Centers  reported 
receiving  external  funding  and  these  funds  accounted  for  0.3  to  2  percent 
of  those  Centers’  funding. 


Education  shapes  the  activities  of  the  R&D  Centers,  Regional  Labs,  and 
Comprehensive  Centers  through  its  funding  documents  and  program 
monitoring.  Education  uses  funding  documents,  such  as  grant 
announcements  and  statements  of  work,  to  guide  and  direct  activities 
included  in  the  mandates  and  to  obtain  help  in  implementing  department 
activities.  In  addition.  Education  assigns  program  officers  to  oversee  the 
activities  of  the  R&D  Centers,  Regional  Labs,  and  Comprehensive  Centers. 

Although  priorities  in  R&D  Center  cooperative  agreements  are  broad  and 
do  not  impose  particular  methods  for  researching  a  topic,  they  are  specific 
enough  to  shape  the  direction  and  breadth  of  the  R&D  Centers’  research 
agenda.  For  example,  in  spelling  out  the  priorities  for  the  R&D  Center  for 
enhancing  young  children’s  development  and  learning.  Education 
identified  topics,  theories,  and  research  areas  that  the  Center  should 
address.  Similarly  in  spelling  out  the  priorities  for  the  R&D  Centers  for 
meeting  the  needs  of  diverse  student  populations.  Education  identified 
topics,  theories,  and  student  populations  to  be  studied. 

In  addition,  the  R&D  Centers’  cooperative  agreements  require  that  a 
portion  of  R&D  Centers’  funds  be  spent  on  tasks  defined  by  OERI.  R&D 
Centers’  cooperative  agreements  require  them  to  reserve  5  percent  of  their 
core  funds  to  carry  out  OERI  initiated  activities  that  assist  OERI  in 
carrying  out  its  responsibilities.  For  example,  OERI  may  require  an  R&D 
Center  to  write  a  briefing  paper  or  conduct  a  research  project. 

Contracts  between  Education  and  the  Regional  Labs  give  the  Regional 
Labs  control  over  most  of  their  activities,  but  also  give  Education  the 
opportunity  to  guide  some  of  their  work.  Regional  Labs  identify  the  critical 
issues  in  their  region  and  develop  plans  to  address  these  issues.  However, 
these  contracts  also  allow  Education  to  assign  each  Regional  Lab  a  broad 
specialty  area  for  example,  early  childhood  education  or  educational 
technology — ^that  reflects  the  national  education  agenda  and  is  aligned 


Education’s  Ability  to 
Shape  R&D  Center, 
Regional  Lab,  and 
Comprehensive  Center 
Agendas  Varies 
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with  a  dominant  theme  of  an  OERI  research  institute  with  which  they  are 
associated/  In  making  Regional  Labs  responsible  for  a  particular  specialty 
area,  the  contracts  required  that  they  (1)  conduct  development,  applied 
research,  and  dissemination  in  that  area;  (2)  keep  abreast  of  developments 
in  their  designated  field;  (3)  provide  subject  area  expertise  to  other  labs; 
and  (4)  work  cooperatively  with  OERI  institutes  as  appropriate.  Although 
Education  defines  the  general  type  of  activities  for  Regional  Labs  in 
specialty  areas,  the  governing  boards  determine  the  focus  of  the  activities 
and  the  extent  to  which  they  met  the  requirements  of  the  contract. 
Additionally,  the  contracts  require  Regional  Labs  to  work  together  on 
areas  of  concern  to  all  Regional  Labs,  such  as  how  to  effectively 
disseminate  their  products  and  develop  a  telecommunications  network. 
Regional  Labs  reported  spending  about  25  percent  of  their  funds  on  these 
required  national  activities  and  on  other  activities  with  a  national  purpose. 
Unlike  the  cooperative  agreements  for  the  R&D  Centers,  Regional  Lab 
contracts  do  not  require  them  to  do  work  to  support  OERI  activities.  Prior 
to  1996,  the  contracts  required  that  the  Regional  Labs  spend  1  percent  of 
their  core  funds  from  OERI  to  support  OERI.  However,  this  requirement 
was  dropped  because  of  objections  from  the  Regional  Labs.  Currently, 
Regional  Labs  may  agree  to  perform  work  for  OERI  in  return  for 
additional  funding. 

Although  the  Comprehensive  Centers  are  not  research  entities.  Education 
shapes  their  activities  by  setting  priorities  for  them  in  funding  documents. 
Such  priorities  include,  for  example,  meeting  with  school  district  officials 
to  review  and  provide  advice  on  district  procedures  for  meeting  federal 
requirements  and  assisting  school  districts  in  the  development  of  student 
assessments.  In  addition.  Comprehensive  Centers  are  required  to  engage 
in  common  activities  identified  by  Education  such  as  conducting  annual 
conferences  on  school  improvement  activities  that  promote  Education’s 
agenda. 

In  addition  to  funding  documents,  program  officers  who  administer  grants 
and  contracts  may  play  an  important  role  in  influencing  activities  of  the 
R&D  Centers  and  Comprehensive  Centers  by  ensuring  that  work 


^The  contracts  awarded  for  December  1995  through  December  2000  contained  tasks  for 
specialty  areas.  The  contracts  awarded  for  December  2000  through  December  2005 
replaced  “specialty  area”  tasks  with  “national  leadership  area”  tasks,  such  as  educational 
leadership  and  teacher  development,  and  required  that  the  leadership  role  include 
synthesizing  research,  disseminating  information,  and  providing  training  to  other  Regional 
Labs. 
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performed  is  consistent  with  work  proposed  and  that  funds  are  being  used 
as  effectively  and  efficiently  as  possible.  Program  officers  are  supposed  to 
perform  the  following  types  of  activities:  (1)  help  to  develop  funding 
documents  and  ensure  that  R&D  Centers  and  Comprehensive  Centers  are 
in  compliance  with  these  documents;  (2)  review  progress  reports,  financial 
reports,  and  products;  and  (3)  approve  dissemination  plans,  staffing 
changes,  and  activities  funded  by  other  sources.  Program  officers  reported 
using  these  oversight  functions  to  ensure  that  the  activities  of  R&D 
Centers  and  Comprehensive  Centers  are  consistent  with  their  proposals. 
For  example.  Education  program  officials  reported  that  they  identified 
activities  that  were  inconsistent  with  those  in  the  R&D  Centers’  proposals 
and  subsequently  negotiated  alternative  activities,  indicated  where 
collaboration  between  some  R&D  Centers  would  be  beneficial,  and 
encouraged  R&D  Centers  to  drop  nonpromising  lines  of  research.  Program 
officers  also  may  play  a  role  in  determining  supplemental  and  future 
funding  decisions. 

Even  though  regional  governing  boards  are  responsible  for  the  oversight 
of  Regional  Labs,  Education’s  program  officers,  nonetheless,  may  still  have 
the  potential  to  influence  the  activities  of  these  labs.  A  program  officer 
assigned  to  a  Regional  Lab  described  their  role  as  that  of  “critical  friends” 
who  use  their  professional  expertise  and  interpersonal  relationships  with 
Regional  Labs’  staff  to  influence  the  activities  of  the  Regional  Labs.  In 
addition.  Regional  Lab  program  officers,  like  R&D  Center  program 
officers,  may  play  a  role  in  determining  supplemental  and  future  funding 
decisions. 

Time  and  authority  may  limit  program  officers’  ability  to  exercise  their 
influence.  In  OERl,  one  program  officer  is  generally  assigned  to  each  R&D 
Center  and  Regional  Lab.  The  program  officers  we  interviewed  reported 
spending  about  50  percent  of  their  time  on  monitoring  activities  related  to 
the  R&D  Centers  and  Regional  Labs  because  of  other  assigned 
responsibilities.  “  Because  the  R&D  Centers  and  Regional  Labs  are 
complex  organizations  and  prolific  producers  of  products  and  services, 
officers  have  to  be  very  selective  in  targeting  their  own  time.  Only  one 
program  officer  is  assigned  to  monitor  all  15  Comprehensive  Centers. 


^"Program  officers  said  they  spend  the  other  60  percent  of  their  time  on  a  wide  range  of 
activities  including  working  on  field  initiated  studies  and  interagency  research  projects, 
writing  statements  of  work  and  cooperative  agreement  announcements,  coordinating  with 
a  number  of  Education  offices,  and  monitoring  other  OERl  programs. 
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Regional  Labs’  Governance 
Structure  Limits 
Education’s  Ability  to 
Provide  Product  Oversight 
and  Shape  Their  Agendas 


The  Congress  has  consistently  given  education  program  oversight  to  either 
a  federal  agency — usually  Education  or  to  the  states,  since  the  states  are 
generally  responsible  for  the  education  of  their  students.  The  Regional 
Labs  are  unlike  other  federal  education  programs  because  neither  the 
federal  government  nor  state  governments  have  oversight  responsibility 
for  programs.  Specifically,  Education  has  little  control  over  Regional  Labs 
because  regional  boards  govern  them.  This  occurs  in  spite  of  the  fact  that 
the  Regional  Labs  get  the  largest  share  of  the  federal  dollars  devoted  to 
these  three  programs  that  conduct  or  support  research.  The  law  requires 
that  the  Regional  Labs  establish  governing  boards  with  regional 
representatives.  The  regional  boards  have  sole  responsibility  for 
determining  the  regional  agenda  and  for  determining  whether  the  Labs  are 
fulfilling  the  responsibility  of  their  contracts,  even  though  these  contracts 
are  funded  by  Education.^  Even  for  the  national  specialty  areas.  Education 
sets  only  wide  parameters  while  the  governing  boards  determine  specific 
activities.  Education  limits  its  communications  with  Regional  Labs  mainly 
to  administrative  issues,  according  to  a  director  we  interviewed  at  a 
Regional  Lab.  In  addition.  Education’s  Regional  Lab  program  officers  told 
us  their  comments  on  Regional  Lab  work  and  products  were  only 
advisory. 


The  law  requires  each  Regional  Lab  to  establish  a  governing  board  that 
reflects  both  a  balanced  representation  of  the  states  in  the  region  and  the 
concerns  of  regional  constituencies  and  includes  teachers  and  education 
researchers.  In  addition.  Education  requires  that  every  chief  state  school 
officer  in  the  area  served  by  the  Regional  Labs  be  offered  an  opportunity 
to  serve  on  the  board  or  to  designate  a  representative.  These  safeguards, 
however,  do  not  assure  that  priorities  of  each  state  in  the  region  will  be 
met.  A  variety  of  factors  potentially  make  achieving  balanced  regional  and 
state  representation  challenging.  First,  Regional  Labs  are  not  given 
guidance  on  how  to  obtain  balance  between  states  and  regional  interests, 
and  regions,  unlike  states,  have  no  formal  governing  body  to  establish 
educational  priorities.  Second,  the  governing  boards,  not  the  states,  have 
the  authority  to  determine  how  members  are  nominated  and  selected. 
Finally,  states  within  regions  vary  substantially  in  size,  population  density, 
poverty  levels,  and  ethnic  composition.  These  factors  may  determine  state 
educational  priorities  in  a  way  that  would  make  them  vary  widely.  For 
example,  California — a  diverse  and  large  state  in  size  and  population — is 


^’Education  provides  financial  oversight,  but  does  not  oversee  the  Regional  Labs’  products 
and  services. 
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included  with  Utah,  Nevada  and  Arizona  in  one  region.  New  York  also  a 
diverse  and  large  state — is  included  with  Vermont  and  other  New  England 
states  in  another  region. 


Collaboration  and 
Coordination  among 
the  R&D  Centers, 
Regional  Labs, 
Comprehensive 
Centers,  and 
Education  Is  Ongoing 


R&D  Centers,  Regional  Labs,  and  Comprehensive  Centers  provided  many 
examples  of  collaborative  and  coordinated  activities.  R&D  Centers, 
Regional  Labs,  and  Comprehensive  Centers  are  required  to  collaborate 
and  coordinate  with  each  other  and  Education.  The  programs  reported 
they  are  most  likely  to  engage  in  collaborative  and  coordinated  activities 
when  they  share  a  common  interest  in  a  specific  student  population,  such 
as  English  language  learners,  or  a  specific  topic,  such  as  assessment. 
Partnerships  between  programs,  common  memberships  in  consortia,  and 
staff  members  who  are  employed  by  more  than  one  of  these  organizations 
facilitate  collaboration  and  coordination  and  help  leverage  resources. 
Education  plays  a  proactive  role  in  promoting  collaboration  and 
coordination  by  including  requirements  for  certain  activities  as  part  of  its 
funding  documents  and  in  its  ongoing  negotiations  with  these 
organizations.  However,  certain  factors  differences  in  student  populations, 
funding  uncertainty,  and  competition  reduce  opportunities  for 
collaboration  and  coordination. 


R&D  Centers,  Regional 
Labs,  and  Comprehensive 
Centers  Collaborate  and 
Coordinate  in  Various 
Ways 


R&D  Centers,  Regional  Labs,  and  Comprehensive  Centers  collaborate  and 
coordinate  with  each  other  and  Education  as  required  by  law  and  the 
documents  that  control  their  funding.  Certain  types  of  collaboration  and 
coordination  flow  naturally  from  overlapping  needs,  interests,  and 
resources.  For  example.  Regional  Labs  and  Comprehensive  Centers  with 
populations  of  students  with  limited  English  proficiency  would  naturally 
make  use  of  the  R&D  Centers,  Regional  Labs,  and  Comprehensive  Centers 
that  have  expertise  in  that  area. 


R&D  Centers,  Regional  Labs,  and  Comprehensive  Centers  reported  a 
variety  of  collaborative  efforts  for  fiscal  year  2000,  including  joint  projects 
and  training.  Joint  projects  included: 


•  Two  Comprehensive  Centers  (the  Northern  California  Comprehensive 
Center  and  the  Southern  California  Comprehensive  Center)  worked 
together  to  produce  teleconferences  to  help  low  performing  schools. 

•  Ten  Regional  Labs  and  an  R&D  Center  (the  National  Center  for  Early 
Development  and  Learning)  produced  a  training  guide  entitled 
Continuity  in  Early  Childhood:  A  Framework  for  Home,  School,  and 
Community  Linkages. 
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•  An  R&D  Center  (the  Center  for  Research  on  the  Education  of  Students 
Placed  At-Risk)  and  a  Regional  Lab  (the  Northwest  Regional  Lab) 
produced  a  joint  publication  about  parent  involvement  in  schools. 

•  A  Regional  Lab  (the  Northwest  Regional  Lab)  and  an  R&D  Center  (the 
Center  for  Research  on  Evaluation,  Standards,  and  Student  Testing) 
created  the  Classroom  Assessment  Tool  Kit. 

Examples  of  training  included: 

•  A  Comprehensive  Center  (the  Southern  California  Comprehensive 
Center)  trained  other  Comprehensive  Centers  to  teach  instructors  how 
to  coach  children  learning  to  read. 

•  An  R&D  Center  (the  Center  for  Research  on  Evaluation,  Standards,  and 
Student  Testing)  trained  a  Regional  Lab  (WestEd)  to  use  a  data 
collection  tool. 

•  An  R&D  Center  (the  National  Center  for  Improving  Student  Learning 
and  Achievement  in  Mathematics  and  Science)  worked  with  a 
Comprehensive  Center  (the  Region  VI  Comprehensive  Center)  to 
conduct  a  professional  development  project  by  teaching  people  to  train 
math  and  science  teachers. 

•  A  Regional  Lab  (the  Southwest  Educational  Development  Laboratory) 
provided  training  to  a  staff  member  of  a  Comprehensive  Center  (the 
Southeast  Comprehensive  Center)  in  the  use  of  Flashlight  and 
Compass:  A  Collection  of  Tools  to  Promote  Instructional  Coherence  a 
tool  for  establishing  teacher  study  groups. 

R&D  Center,  Regional  Lab,  and  Comprehensive  Center  staff  also  provided 

many  examples  of  coordination  efforts  meant  to  ensure  that  each  was 

aware  of  the  others’  projects.  Some  examples  include  the  following: 

•  As  a  follow-up  to  Education’s  National  Awards  for  Model  Professional 
Development,  three  Regional  Labs,  WestEd,  the  North  Central  Regional 
Lab,  and  the  Mid-Continent  Regional  Lab,  studied  how  the  awarded 
districts  supported  districtwide  teacher  and  student  learning.  An  R&D 
Center,  the  Center  for  the  Study  of  Teaching  and  Policy,  shared  its  data 
on  resource  allocation  among  some  of  the  same  districts  and 
contributed  to  the  research  questions  and  design.  The  Regional  Labs 
shared  the  findings  with  Education. 

•  An  R&D  Center,  Center  for  Research  on  Education,  Diversity,  and 
Excellence,  reviewed  work  on  Spanish  Writing  Assessment  done  by  a 
Regional  Lab,  Northwest  Regional  Lab. 
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Legislative  Requirements 
and  Common  Interests 
Foster  Collaboration  and 
Coordination 


Legislation  requires  that  the  Regional  Labs  and  Comprehensive  Centers 
collaborate  and  coordinate  with  each  other  and  with  the  R&D  Centers,  but 
their  mandates  differ  in  the  amount  of  collaboration  and  coordination  they 
require.  The  Regional  Labs  are  required  to  collaborate  and  coordinate  with 
each  other,  Education-funded  technical  assistance  providers,  and  OERl 
institutes,  and  to  share  and  plan  joint  activities  with  other  Education- 
funded  state,  and  federal  programs.  They  are  also  required  to  establish  a 
network  for  sharing  information,  planning  activities  involving  multiple 
regions,  and  working  on  national  projects.  The  Comprehensive  Centers  are 
required  to  share  information,  coordinate  services,  and  work 
cooperatively  with  the  Regional  Labs,  R&D  Centers,  Education’s  regional 
offices,  state  and  local  educational  agencies,  and  all  other  Education- 
funded  research,  development,  dissemination,  and  technical  assistance 
programs.  The  R&D  Centers  do  not  have  specific  legislative  requirements 
to  collaborate  and  coordinate  with  other  Education-funded  programs; 
however,  they  are  required  to  do  so  in  their  cooperative  agreements. 


R&D  Centers,  Regional  Labs,  and  Comprehensive  Centers  collaborate  and 
coordinate  when  they  have  an  interest  in  the  same  student  population.  The 
following  examples  illustrate  how  student  populations  provide  a  focal 
point  for  collaboration  and  coordination. 


•  A  member  of  the  Mid-Continent  Regional  Lab,  which  includes  states 
with  large  numbers  of  Native  Americans,  sat  on  the  steering  committee 
of  the  National  Research  Center  on  the  Gifted  and  Talented,  an  R&D 
Center.  This  committee  oversaw  the  production  of  a  publication  on 
talented  American  Indian  and  Alaskan  Native  students. 

•  Two  R&D  Centers,  the  Center  for  Research  on  Education,  Diversity, 
and  Excellence  and  the  National  Center  for  Improving  Student 
Learning  and  Achievement  in  Mathematics  and  Science,  published  a 
newsletter  on  issues  related  to  how  diverse  students  learn  math  and 
science. 

•  All  15  of  the  Comprehensive  Centers  created  an  Internet  mailing  list 
about  English-language  learners  to  share  information  from  their 
regions,  identify  staff  with  proficiency  in  meeting  the  needs  of  English- 
language  learners,  and  disseminate  information. 


Interlacing  Organizational 
Relationships  Generally 
Facilitate  Collaboration 
and  Coordination 


The  structures  of  the  R&D  Centers,  Regional  Labs,  and  Comprehensive 
Centers  foster  collaboration  and  coordination  among  them  and  other 
entities.  In  some  cases,  partnerships  may  encourage  collaboration.  In 
other  cases,  collaboration  occurs  because  one  single  entity  operates  both 
an  R&D  Center,  Regional  Lab,  and/or  Comprehensive  Center. 
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Partnerships  may  encourage  collaboration  by  establishing  bridges 
between  programs.  R&D  Centers,  Regional  Labs,  and  Comprehensive 
Centers  are  programs  that  operate  through  parent 
organizations  universities,  not-for-profit  organizations,  and  educational 
agencies.  These  parent  organizations  create  consortia — formal 
partnerships  with  other  universities,  not-for-profit  organizations,  for-profit 
corporations,  and  educational  agencies — to  run  R&D  Centers  and 
Comprehensive  Centers.  Participants  in  partnerships  that  run  R&D 
Centers  range  from  2  universities  to  29  universities  and  not-for-profit 
organizations.  Figure  8  shows  one  R&D  Center  that  is  a  consortium  of  5 
universities  and  the  affiliations  they  have  with  other  R&D  Centers. 
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Figure  8:  Organizational  Structure  of  an  R&D  Center  and  Its  Affiliations  with  Other  R&D  Centers 
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Unlike  R&D  Centers  and  Comprehensive  Centers,  the  parent  organizations 
that  run  Regional  Labs  do  not  form  partnerships  with  other  universities, 
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not-for-profits,  or  educational  agencies  to  run  the  Regional  Labs.  They 
may,  however,  be  asked  by  a  Comprehensive  Center  or  R&D  Center  to 
enter  into  a  partnership  in  order  to  provide  specialized  services.  We  found 
four  parent  organizations  that  operated  Regional  Labs  that  had  formed 
partnerships  with  Comprehensive  Centers.  Figure  9  shows  an  example  of 
such  a  partnership. 
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Figure  9:  Regional  Lab  Parent  Organization,  In-House  Programs,  and  Services  Provided  to  Partners 
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Participants  in  partnerships  that  run  Comprehensive  Centers  range  from 
three  to  eight  universities,  not-for-profit  organizations,  and  educational 
agencies.  Most  parent  organizations  that  run  Comprehensive  Centers 
partner  with  other  organizations  to  obtain  expert  services  in  specialized 
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areas,  such  as  migrant  education  or  Indian  education.  Because  each  of 
these  specialized  partners  may  work  with  as  many  as  four  Comprehensive 
Centers,  these  partnerships  may  establish  bridges  that  forge  coordination 
in  particular  topical  areas.  For  example,  ESCORT,  a  former  categorical 
technical  assistance  center,  specializes  in  migrant  education  and  partners 
with  four  Comprehensive  Centers  to  provide  services  in  that  area. 

Collaboration  also  results  when  a  single  organization  operates  both  an 
R&D  Center,  Regional  Lab,  or  a  Comprehensive  Center.  For  example,  staff 
from  the  Western  Regional  Lab  and  the  Northern  California 
Comprehensive  Center  participated  in  a  workgroup  that  developed  a  guide 
on  how  schools  could  better  obtain  student  perspectives  and  suggestions 
to  improve  school  planning.  As  shown  in  figure  9,  both  the  Western 
Regional  Lab  and  the  Northern  California  Comprehensive  Center  are  run 
by  the  same  parent  organization.  Five  of  the  10  parent  organizations  that 
run  a  Regional  Lab  also  run  a  Comprehensive  Center.  One  of  the  12  parent 
organizations  that  operates  an  R&D  Center  also  operates  a  Comprehensive 
Center. 

These  interlacing  organizational  relationships  allow  R&D  Centers, 

Regional  Labs,  and  Comprehensive  Centers  to  leverage  resources.  For 
example,  an  Education  official  working  with  the  Comprehensive  Centers 
told  us  that  through  their  parent  organizations  and  partnerships 
Comprehensive  Centers  are  able  to  leverage  the  personnel  and  expertise 
needed  to  perform  their  work.  In  those  cases  where  parent  organizations 
run  two  programs,  we  were  told  that  staff  divide  their  time  between 
programs  to  leverage  expertise.  Similarly,  a  director  of  a  parent 
organization  that  runs  a  Regional  Lab  and  a  Comprehensive  Center  stated 
that  the  Comprehensive  Center  draws  upon  experts  assigned  primarily  to 
other  projects  to  obtain  skills  needed  to  implement  particular  activities. 

Individual  relationships,  such  as  staff  members  holding  multiple 
appointments  within  R&D  Centers,  Regional  Labs,  or  Comprehensive 
Centers  also  facilitate  collaboration  and  coordination  among  R&D 
Centers,  Regional  Labs,  and  Comprehensive  Centers.  For  example,  a 
principal  investigator  for  the  Consortium  for  Policy  Research  in 
Education,  an  R&D  Center,  also  works  on  projects  for  the  National  Center 
for  Improving  Student  Learning  and  Achievement  in  Mathematics  and 
Science,  another  R&D  Center. 
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Education  Plays  a 
Proactive  Role  in 
Encouraging  Collaboration 
and  Coordination 


The  funding  agreements  between  Education  and  the  R&D  Centers, 
Regional  Labs,  and  Comprehensive  Centers  reflect  Education’s 
interpretation  and  implementation  of  legislative  requirements  for 
collaboration  and  coordination.  The  cooperative  agreements  for  the  R&D 
Centers  require  them  to  collaborate  and  coordinate  with  Regional  Labs, 
Comprehensive  Centers,  other  federal  programs,  policy  institutions,  and 
advocacy  groups.  R&D  Centers  also  agree,  in  their  funding  documents,  to 
conduct  an  annual  meeting  to  share  research  with  Education  and  other 
research  and  development  programs,  and  collaborate  with  OERl.  Regional 
Lab  contracts  with  Education  require  that  they  participate  in  at  least  two 
meetings  a  year  convened  to  discuss  issues  related  to  Education-funded 
programs.  In  addition.  Education’s  contracts  for  Regional  Labs  require 
Regional  Lab  representatives  to  meet  with  OERl  annually  and 
chairpersons  of  the  governing  boards  to  meet  with  OERl  when  their 
contract  begins.  Regional  Labs  also  have  an  option  in  their  contracts  with 
Education  that  allows  them  to  earn  supplemental  funds  by  agreeing  to 
perform  work  in  collaboration  and  coordination  with  OERl,  including 
sponsoring  meetings  and  panels  and  writing  briefs.  Cooperative 
agreements  for  the  Comprehensive  Centers  require  them  to  meet  with 
seven  different  Education-funded  programs,  including  the  Regional  Labs, 
to  discuss  collaboration  and  coordination;  plan  a  national  conference; 
engage  in  a  common  project  to  improve  teaching;  and  collaborate  with 
each  other  and  local  and  state  educational  agencies. 

Cooperative  agreements  for  the  R&D  Centers  and  Comprehensive  Centers 
also  outline  Education’s  responsibilities  to  facilitate  collaboration  and 
coordination.  For  example,  the  funding  agreement  for  the  Comprehensive 
Centers  specifies  that  Education  officials  will  work  with  Comprehensive 
Centers  in  planning  conferences  and  identifying  areas  for  collaboration 
and  coordination.  The  funding  agreements  that  Education  has  with 
Regional  Labs  do  not  outline  Education’s  responsibilities  for  facilitating 
collaboration  and  coordination. 

Education  officials  may  also  identify  appropriate  areas  for  collaboration 
and  coordination.  For  example,  an  institute  director  told  us  that  she 
contacted  an  R&D  Center  in  another  institute  to  discuss  possible  areas  for 
collaboration  and  coordination  with  the  R&D  Center  in  her  institute. 
Similarly,  some  Comprehensive  Center  officials  stated  that  the  program 
officer  assigned  to  them  identified  areas  for  cross-program  collaboration 
and  coordination  and  communicated  with  them  frequently.  For  example, 
the  program  officer  suggested  collaborating  and  coordinating  on  the 
creation  of  a  common  framework  for  working  with  low  performing 
schools  in  Comprehensive  Center  regions. 
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Differences  in  Student 
Population  and  Topics, 
Funding  Uncertainty,  and 
Competition  Reduce 
Opportunities  for 
Collaboration  and 
Coordination 


Despite  the  efforts  made  by  the  R&D  Centers,  Regional  Labs, 
Comprehensive  Centers,  and  Education  in  fostering  collaboration  and 
coordination,  barriers  exist.  R&D  Centers,  Regional  Labs,  and 
Comprehensive  Centers  cited  differences  in  student  populations  and 
topics,  uncertain  funding,  and  competition  as  hindrances  to  collaboration 
and  coordination.  R&D  Centers  have  different  research  focuses  and 
conduct  research  on  different  topics.  Regional  Labs  and  Comprehensive 
Centers  serve  diverse  geographical  areas  with  different  interests.  These 
differences  potentially  reduce  opportunities  for  collaboration  and 
coordination.  For  example,  an  R&D  Center  with  a  focus  on  postsecondary 
education  would  have  little  or  no  reason  to  collaborate  or  coordinate  on 
substantive  issues  with  other  R&D  Centers,  Regional  Labs,  and 
Comprehensive  Centers  that  focus  on  research  related  to  elementary  and 
secondary  education.  Funding  uncertainties  also  make  collaboration  and 
coordination  difficult.  Directors  said  they  were  often  reluctant  to  write 
collaborative  activities  into  their  proposals  because  they  did  not  know 
which  organizations  would  win  future  funding  competitions.  Competition 
also  limits  collaboration  and  coordination.  The  education  research  and 
technical  assistance  business  is  a  competitive  industry.  Like  others  in 
competitive  industries,  R&D  Center,  Regional  Lab,  and  Comprehensive 
Center  staff  are  protective  of  ideas  and  practices  that  give  them 
advantages  over  other  organizations  that  they  perceive  as  competitors  for 
future  sources  of  funding. 


Education  Needs  to 
Improve  Its 
Assessments  of  the 
R&D  Centers, 
Regional  Labs,  and 
Comprehensive 
Centers 


The  recently  funded  evaluations  of  the  R&D  Centers,  Regional  Labs,  and 
Comprehensive  Centers  provided  limited  information  on  outcomes  of  the 
activities  conducted  by  these  programs.  OERl  is  required  to  use  peer 
review  to  evaluate  the  R&D  Centers  and  Regional  Labs.  Peer  review  is  well 
accepted  and  widely  used  for  assessing  the  merit  of  research  proposals 
and  the  scientific  soundness  of  research.  Unlike  the  R&D  Centers, 
research  is  only  a  relatively  small  part  of  what  Regional  Labs  do.  Their 
other  activities  dissemination  and  technical  assistance  would  have  been 
more  appropriately  evaluated  using  methods  other  than  peer  review.  The 
peer  review  processes  that  Education  used  to  evaluate  the  R&D  Centers 
and  Regional  Labs  have  shortcomings  that  limited  the  usefulness  of  the 
findings.  First,  the  peer  review  process  used  has  the  potential  for  bias 
because  the  R&D  Centers  and  the  Regional  Labs  selected  most  of  the 
products  that  were  reviewed.  Second,  the  processes  were  cumbersome. 
For  example.  Education  required  each  member  of  a  review  team  to  write 
an  individual  assessment  report  and  to  review  all  contracts,  contract 
modifications,  progress  reports,  and  annual  updates  for  a  three-year 
period.  In  addition,  with  regard  to  the  Comprehensive  Centers,  the 
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customer  satisfaction  survey  evaluations  of  the  Comprehensive  Centers 
did  not  provide  information  on  individual  centers. 


Recent  Peer  Reviews  of 
the  R&D  Centers  and 
Regional  Labs  Provided 
Limited  Information 


Traditionally,  Education  has  used  peer  review  to  evaluate  activities  carried 
out  by  OERI,  including  those  conducted  by  the  R&D  Centers  and  the 
Regional  Labs.  The  Educational  Research,  Development,  Dissemination, 
and  Improvement  Act  of  1994  codified  this  practice  by  requiring  OERI  to 
develop  peer  review  standards  for  evaluating  and  assessing  the 
performance  of  recipients  of  grants,  cooperative  agreements,  and 
contracts  that  exceed  $100,000,  as  well  as  for  selecting  proposals  for 
funding  and  identifying  exemplary  and  promising  educational  programs. 

Historically,  peer  review  has  been  used  extensively  in  the  selection  of 
proposed  research  projects  and,  to  a  lesser  extent,  to  evaluate  research 
and  development  programs.  Peer  review  entails  an  independent 
assessment  of  the  technical  or  scientific  merit  of  research  by  peers  who 
are  scientists  with  knowledge  and  expertise  equal  to  that  of  the 
researchers  whose  work  they  review.  It  is  sometimes  used  to  evaluate 
research  when  the  ultimate  outcomes  of  the  research  are  unpredictable. 

Peer  review  may  be  appropriate  for  evaluating  some  activities  conducted 
by  the  R&D  Centers  but  other  evaluation  techniques  are  better  suited  for 
evaluating  the  many  activities  of  the  R&D  Centers  and  Regional  Labs. 
According  to  the  National  Academy  of  Sciences,  although  peer  review  is 
well  suited  for  assessing  theory-driven  research  with  potential  long-term 
effects,  it  is  less  appropriate  for  assessing  applied  research,  development, 
technical  assistance,  and  dissemination  efforts  in  which  outcomes  are 
anticipated  and  can  be  measured  over  a  relatively  short  period  of  time.'^ 
The  R&D  Centers  and  Regional  Labs  engage  in  many  research  activities 
that  are  designed  to  achieve  practical  outcomes.  Evaluation  methods  that 
measure  outcomes  customer  surveys,  comparisons  with  similar  programs, 
and  controlled  case  studies  may  be  better  suited  for  evaluating  these 
activities. 


^^National  Academy  of  Sciences,  Committee  on  Science,  Engineering,  and  Public  Policy, 
Evaluating  Federal  Research  Programs  (Washington,  D.C.:  National  Academy  Press, 
1999). 
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Education’s  peer  review  processes  have  the  potential  for  bias  and  were 
cumbersome.  Both  of  these  conditions  limited  the  usefulness  of  findings  in 
addressing  key  issues. 

•  The  self-selection  of  materials  for  review  by  R&D  Centers  and  Regional 
Labs  did  not  provide  a  representative  cross  section  of  their  products 
and  services.  R&D  Center  staff  were  involved  in  deciding  which 
products  were  to  be  reviewed  and  each  Regional  Lab  nominated  five  or 
six  major  products  or  services,  two  of  which  were  selected  for  review. 
While  this  approach  allowed  reviewers  an  in-depth  look  at  major 
program  initiatives,  it  did  not  provide  reviewers  with  a  cross  section  of 
the  R&D  Centers  or  Regional  Labs’  work,  nor  did  it  allow  them  to 
generalize  from  these  works  to  other  activities. 

•  Selecting  as  reviewers  for  the  R&D  Centers  and  Regional  Labs  some 
individuals  who  have  previouslv  evaluated  the  merits  of  the  grant 
applications  or  proposals  raised  questions  about  obiectivitv.  It  is  likely 
that  the  individuals  who  selected  these  organizations  as  grant  or 
contract  recipients  might  want  their  original  choices  validated. 

•  Requiring  peers  to  have  a  broad  mix  of  skills  made  selection  and 
scheduling  of  reviewers  difficult.  Unlike  other  agencies  that  select 
peers  solely  on  the  basis  of  their  expertise  in  the  area  of  work,  OERl 
requires  that  review  panels  include  individuals  with  a  broad  range  of 
knowledge  and  experience.  For  example,  OERl  standards  require  peer 
review  panels  to  include  individuals  with  in-depth  knowledge  of 
education  policy  or  practice  and  in-depth  knowledge  of  theories  and 
methods  of  study  related  to  the  subject  area.  These  requirements 
complicated  the  identification  of  peer  review  panels. 

•  The  amount  of  material  to  be  reviewed  was  extensive  in  terms  of  the 
short  time  frames  of  the  assessments  and  the  complexity  of  the 
organizations.  Over  a  short  period  of  time — for  example,  5  days  on  site 
for  the  review  of  a  Regional  Lab,  with  half  that  time  devoted  to  data 
gathering  and  the  other  half  to  writing  the  reports — all  reviewers  were 
required  to  read  immense  amounts  of  material,  including  funding 
documents,  statements  of  work,  proposals,  and  progress  reports,  to 
learn,  in  detail,  how  the  programs  performed  their  work  and  to  write 
individual  reports.  Some  reviewers  complained  that  they  only  had  time 
to  “scratch  the  surface,”  and  that  much  of  the  material  they  had  to 
review  was  repetitive.  If  responsibilities  could  have  been  shared,  peer 
reviewers  would  have  been  able  to  gather  more  in-depth  knowledge. 
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•  In  some  cases,  materials  chosen  for  review  by  the  R&D  Centers  were 
incomplete.  Reviewers  of  some  programs  noted  that  materials 
addressing  the  rationale,  hypotheses,  controls,  and  usefulness  of 
studies  were  often  insufficient  for  them  to  make  informed  judgements. 

•  Assessments  took  place  midway  in  the  funding  cycles.  The  reviews 
generally  took  place  during  the  third  year  of  a  5-year  contract.  As  a 
result,  reviewers  were  hesitant  to  be  critical  in  reports  because  many 
studies  were  on  going  and  therefore  could  improve  by  the  end  of  the 
contract. 


•  The  dual  purposes  of  the  peer  reviews  inhibited  candor.  On  one  hand. 
Education  designed  the  reviews  to  be  formative  evaluations — 
evaluations  that  were  to  focus  on  the  performance  of  the  programs  in 
terms  of  their  missions  and  the  technical  quality  of  their  products.  In 
this  regard,  reviews  were  to  examine  the  overall  quality  of  the  work  of 
the  R&D  Centers  and  Regional  Labs,  the  extent  to  which  R&D  Centers 
and  Regional  Labs  performed  their  work  on  time  and  met  professional 
standards.  However,  the  reviews  were  also  designed  to  assess  the 
usefulness,  outcomes,  and  effects  of  their  work  to  help  OERI  determine 
if  the  R&D  Centers  and  Regional  Labs  merited  continued  funding.  The 
peer  review  process  depended  exclusively  upon  expert  opinion  rather 
than  directly  measuring  how  useful  the  research  was  or  its  effects. 
Reviewers  discussed  “the  potential”  value  of  activities  and  were  not 
able  to  predict  the  ability  of  the  entities  to  contribute  substantially  to 
the  field.  Moreover,  they  did  not  believe  their  findings  should  have 
influence  over  funding  decisions,  which  affected  what  they  wrote  in 
their  reports. 


Evaluation  of  the 
Comprehensive  Centers 
Did  Not  Provide 
Comparable  Information 
on  Individual  Centers 


Education’s  evaluation  of  the  Comprehensive  Centers  met  the 
requirements  of  the  law  but  provided  little  information  that  would  help 
Education  determine  if  each  Comprehensive  Center  was  meeting  the 
needs  of  its  customers.  The  Elementary  and  Secondary  Education  Act 
requires  the  Secretary  of  Education  to  collect  information  about  the 
availability  and  quality  of  services  provided  by  the  Comprehensive  Centers 
and  to  conduct  surveys  to  determine  if  populations  served  by  the 
Comprehensive  Centers  are  satisfied  with  their  access  to  services  and  the 
quality  of  those  services.  As  part  of  the  year  2000  evaluation,  a  contractor 
surveyed  the  satisfaction  of  customers  who  had  participated  in  either  of 
two  activities  offered  by  a  Comprehensive  Center.  These  activities  were 
selected  from  among  all  the  activities  offered  by  the  Comprehensive 
Center  because  they  were  long-term  or  intensive.  The  contractors 
surveyed  customers  by  randomly  selecting  them  from  a  list  prepared  by 
each  Comprehensive  Center.  The  contractor  also  surveyed  a  nationally 
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representative  sample  of  state  and  district  officials  that  they  identified  as  a 
likely  pool  of  customers  for  the  centers  and  a  sample  of  gatekeepers — 
individuals  that  had  requested  or  negotiated  for  services  on  behalf  of  their 
school  or  school  district.  In  randomly  selecting  customers  to  survey,  the 
contractor  did  not  choose  a  number  large  enough  from  each 
Comprehensive  Center’s  list  to  allow  any  reliable  generalizations  to  be 
made  about  a  particular  Comprehensive  Center.  Likewise,  the  sample  of 
potential  customers  and  gatekeepers  was  not  suitable  to  draw  conclusions 
about  an  individual  Comprehensive  Center.  The  inclusion  of 
representative  activities  and  customers  for  each  Comprehensive  Center 
would  have  increased  the  cost  of  the  mail  survey.  However,  not  drawing  a 
sample  that  was  representative  at  the  center  level  reduced  the  usefulness 
of  the  evaluation  because  Education  could  not  identify  variation  across 
Comprehensive  Centers  or  obtain  information  to  improve  practices  at 
individual  Comprehensive  Centers. 


Conclusions 


The  Regional  Labs  are  unlike  most  federal  education  programs  because 
neither  the  federal  government  nor  state  governments  have  oversight 
responsibility  for  their  programs.  Under  the  current  structure.  Education 
is  accountable  for  the  activities  it  funds  through  the  Regional  Labs,  but 
current  laws  limit  its  ability  to  oversee  those  activities.  Not  only  is  federal 
oversight  limited,  states  also  have  limited  control  over  the  regional  agenda. 
Although  the  requirement  that  governing  boards  have  a  balanced 
representation  of  states  in  the  region  may  ensure  state  input  for  the 
agenda  of  the  Regional  Lab,  the  regional  priorities  that  the  Regional  Labs 
serve  may  not  correspond  to  the  educational  priorities  of  all  states  in  the 
regions.  Thus,  neither  the  states  nor  Education  can  ensure  that  the 
Regional  Labs  are  meeting  the  needs  of  the  states. 

Congressionally  mandated  peer  reviews  of  the  R&D  Centers  and  Regional 
Labs  have  produced  limited  information  about  the  overall  performance  of 
each  organization,  the  services  they  provide,  or  the  extent  to  which 
teaching  and  learning  are  improved  by  the  products  R&D  Centers  and 
Regional  Labs  produce.  In  part,  this  was  because  reviewers  could  not 
divide  the  tasks  among  themselves,  as  each  reviewer  was  required  to  do  a 
full,  independent  assessment.  In  addition,  given  the  present  practice 
allowing  the  Regional  Lab  directors  to  choose  products  and  services  for 
review,  the  potential  exists  for  bias  and  therefore  calls  into  question  the 
quality  of  the  assessments.  As  a  result.  Education  lacks  information  that 
would  be  useful  in  making  funding  decisions  or  improving  the 
performance  of  each  organization.  Unless  standard  program  evaluation 
techniques,  such  as  customer  surveys  or  controlled  case  studies,  are 
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introduced  into  the  evaluation  process  of  these  entities,  these  problems 
are  likely  to  continue. 

The  value  of  the  mandated  year  2000  evaluations  of  the  Comprehensive 
Centers  was  limited.  We  recognize  that  addressing  this  problem  would 
involve  expanding  the  sample  sizes  and,  for  the  consumer  survey,  the 
number  of  activities  selected.  However,  currently,  neither  the 
Comprehensive  Centers  themselves  nor  Education  can  determine  if  the 
customers  of  a  particular  Comprehensive  Center  are  satisfied  with  some 
or  all  of  its  products  and  services.  As  a  result,  problems  at  any  given  center 
could  go  unchecked.  Moreover,  Education  cannot  assess  the  relative 
strengths  and  weakness  of  individual  Comprehensive  Centers  so  it  can 
improve  the  services  in  all  centers  and  make  better  funding  decisions. 


Matters  for 

Congressional 

Consideration 


If  the  Congress  wishes  to  ensure  greater  accountability  to  a  governmental 
entity  for  the  Regional  Labs,  it  could  consider  either  giving  Education 
responsibility  for  determining  the  regional  agenda  and  overseeing  the 
products  and  services  of  the  Regional  Labs  or  Education  could  provide 
these  funds  to  states,  possibly  as  part  of  a  larger  formula  grant,  for 
subsequent  distribution  by  each  state.  This  would  give  states  greater 
control  in  purchasing  research-based  educational  products  and  services. 

If  the  Congress  wants  to  increase  the  usefulness  of  the  assessments  of  the 
R&D  Centers  and  Regional  Labs,  the  Congress  should  consider  allowing 
Education  to  use  methods  other  than  peer  review  when  such  methods  are 
more  appropriate  than  peer  review  for  evaluating  the  activities  of  R&D 
Centers  and  Regional  Labs. 


Recommendations  for 
Executive  Action 


To  improve  the  assessments  of  the  R&D  Centers  and  Regional  Labs,  we 
recommend  that  the  Secretary  of  Education  direct  the  Assistant  Secretary 
of  OERl  to 


•  use  random  selection  of  projects,  services  and  products  to  be  reviewed 
when  conducting  future  evaluations  of  R&D  Centers  and  Regional  Labs, 
and 

•  revise  the  peer  review  standards  to  allow  for  division  of  labor  and  greater 
concentration  on  assessing  the  quality  of  projects,  services  and  products. 

To  improve  the  performance  and  usefulness  of  the  Comprehensive 
Centers,  the  Secretary  of  Education  should  direct  the  Assistant  Secretary 
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of  Elementary  and  Secondary  Education  and  the  Assistant  Secretary  of 
Bilingual  Education  and  Minority  Languages  Affairs  to 

•  design  future  evaluations  of  the  Comprehensive  Centers  to  produce 
findings  pertaining  to  individual  Comprehensive  Centers. 

Agency  Comments 

We  provided  a  draft  of  this  report  to  the  Department  of  Education  for 
comment.  Education’s  Executive  Secretariat  confirmed  by  e-mail  that 
Department  officials  had  reviewed  the  draft  and  had  no  comments  except 
for  a  few  technical  clarifications  regarding  funding. 

We  are  sending  copies  of  this  report  to  the  Secretary  of  Education,  the 
appropriate  congressional  offices,  and  other  interested  parties.  If  you  have 
any  questions,  please  contact  me  at  (202)  512-7015  or  Eleanor  L.  Johnson 
(202)  512-7209.  Other  contributors  can  be  found  in  appendix  I. 


Mamie  S.  Shaul 

Director,  Education,  Workforce, 
and  Income  Security  Issues 


Page  39 


GAO-02-190  Education  Research 


Appendix  I:  GAO  Contacts  and  Staff 
Acknowledgments 


GAO  Contacts 


Eleanor  L.  Johnson  (202)  512-7209 
Kathleen  D.  White  (202)  512-8512 


Acknowledgments 


In  addition  to  those  named  above,  Malcolm  Brewery,  Jr.,  Tahra  N. 
Edwards,  Richard  B.  Kelley,  and  Sarah  Moorhead  made  key  contributions 
to  this  report. 


(102010) 


Page  40 


GAO-02-190  Education  Research 


GAO’S  Mission 

The  General  Accounting  Office,  the  investigative  arm  of  Congress,  exists  to 
support  Congress  in  meeting  its  constitutional  responsibilities  and  to  help 
improve  the  performance  and  accoimtability  of  the  federal  government  for  the 
American  people.  GAO  examines  the  use  of  public  funds;  evaluates  federal 
programs  and  policies;  and  provides  analyses,  recommendations,  and  other 
assistance  to  help  Congress  make  informed  oversight,  policy,  and  fimding 
decisions.  GAO’s  commitment  to  good  government  is  reflected  in  its  core  values 
of  accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  is  through  the 
Internet.  GAO’s  Web  site  (www.gao.gov)  contains  abstracts  and  full-text  files  of 
current  reports  and  testimony  and  an  expanding  archive  of  older  products.  The 

Web  site  features  a  search  engine  to  help  you  locate  documents  using  key  words 
and  phrases.  You  can  print  these  documents  in  their  entirety,  including  charts  and 
other  graphics. 

Each  day,  GAO  issues  a  list  of  newly  released  reports,  testimony,  and 
correspondence.  GAO  posts  this  list,  known  as  “Today’s  Reports,”  on  its  Web  site 
daily.  The  list  contains  links  to  the  full-text  document  files.  To  have  GAO  e-mail 
this  list  to  you  every  afternoon,  go  to  www.gao.gov  and  select  "Subscribe  to  daily 
e-mail  alert  for  newly  released  products"  under  the  GAO  Reports  heading. 

Order  by  Mail  or  Phone 

The  first  copy  of  each  printed  report  is  free.  Additional  copies  are  $2  each.  A 
check  or  money  order  should  be  made  out  to  the  Superintendent  of  Documents. 
GAO  also  accepts  VISA  and  Mastercard.  Orders  for  100  or  more  copies  mailed  to  a 
single  address  are  discoimted  25  percent.  Orders  should  be  sent  to: 

U.S.  General  Accoimting  Office 

P.O.  Box  37050 

Washington,  D.C.  20013 

To  order  by  Phone:  Voice:  (202)  512-6000 

TDD:  (202)  512-2537 

Fax:  (202)  512-6061 

Visit  GAO’s  Document 
Distribution  Center 

GAO  Building 

Room  1100,  700  4th  Street,  NW  (comer  of  4th  and  G  Streets,  NW) 

Washington,  D.C.  20013 

To  Report  Fraud, 

Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm. 

E-mail:  fraudnet@gao.gov,  or 

1-800-424-5454  or  (202)  512-7470  (automated  answering  system). 
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